Papers with calibration methods
Bag of Tricks for In-Distribution Calibration of Pretrained Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models (PLMs) often predict over-confidently. |
| Approach: | They propose to use ensemble learning and data augmentation to improve confidence calibration for PLMs by combining calibration techniques with a trade-off between accuracy and classification. |
| Outcome: | The proposed calibration method improves classification accuracy and confidence in pre-trained language models by combining several calibration techniques. |
A Close Look into the Calibration of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. |
| Approach: | They conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. |
| Outcome: | The proposed methods significantly reduce PLMs’ confidence in wrong predictions. |
Reconfidencing LLMs from the Grouping Loss Perspective (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to calibrate confidence scores for large language models often overlook biases towards certain groups, such as specific nationalities. |
| Approach: | They propose a method to calibrate confidence scores of Large Language Models by considering different groups, a process they call reconfidencing. |
| Outcome: | The proposed method mitigates biases against minority groups, the authors show . they show that the proposed method is more reliable than existing methods . |
Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models suffer from severe miscalibration for both in-distribution and out-of-difference data due to over-parameterization. |
| Approach: | They propose a regularized method to improve in-distribution and out-of-distance calibrations by using on-manifold regularization and off-manfold regularisation. |
| Outcome: | The proposed method outperforms existing methods for text classification in terms of expectation calibration error, misclassification detection, and OOD detection on six datasets. |
Generative Calibration for In-context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning is one of the most exciting features of large language models . performance is sensitive to various configurations of the prompt, such as the choice or order of the training examples. |
| Approach: | They propose to calibrate the in-context predictive distribution by adjusting the label marginal . they find that the proposed method outperforms the ICL and state-of-the-art calibration methods . |
| Outcome: | The proposed method outperforms state-of-the-art methods by 27% absolute in macro-F1. |
VSCBench: Bridging the Gap in Vision-Language Model Safety Calibration (2025.findings-acl)
Copied to clipboard
Jiahui Geng, Qing Li, Zongxiong Chen, Yuxia Wang, Derui Zhu, Zhuohan Xie, Chenyang Lyu, Xiuying Chen, Preslav Nakov, Fakhri Karray
| Challenge: | Existing safety calibration methods focus on model undersafety, where the model responds to hazardous queries, while neglecting oversafetiness, where models refuse to answer safe queries. |
| Approach: | They propose safety calibration which addresses both undersafety and oversafetiness by comparing model responses to a novel dataset of 3,600 image-text pairs. |
| Outcome: | The proposed methods have been used to evaluate safety calibration across image-centric and text-centric scenarios. |
Extracting Chemical-Protein Interactions via Calibrated Deep Neural Network and Self-training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Several natural language processing methods have been used to extract interactions between chemicals and proteins from biomedical text data. |
| Approach: | They propose a method to extract chemical–protein interactions from biomedical text data . they use a pre-trained language-understanding model and calibration techniques to estimate uncertainty . |
| Outcome: | The proposed approach achieves state-of-the-art performance on the Biocreative VI ChemProt task while preserving higher calibration abilities. |
Re-Examining Calibration: The Case of Question Answering (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing calibration methods do not provide significant gains in accuracy. |
| Approach: | They propose a new calibration metric that better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions. |
| Outcome: | The proposed calibration method better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions. |
Selectively Answering Visual Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are capable of visual question answering (VQA) with unprecedented accuracy. |
| Approach: | They propose a calibration score that can be used to quantify uncertainty in visual question answering models. |
| Outcome: | The proposed calibration score is better calibrated than in text-only models for in-context learning. |
Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for text quality evaluation. |
| Approach: | They propose two methods to improve the accuracy of LLM evaluators by Bayesian inference. |
| Outcome: | The proposed methods improve the accuracy of the win rate estimation using LLMs . the proposed methods are based on six datasets covering story generation, summarization, and instruction following tasks . |
On the Calibration of Massively Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Massively Multilingual Language Models (MMLMs) have gained popularity due to their effectiveness in cross-lingual transfer. |
| Approach: | They investigate how well calibrated MMLMs are with respect to confidence . they find that calibration methods like temperature scaling and label smoothing improve calibration . |
| Outcome: | The proposed models are able to generalize in languages unseen during fine-tuning, but they are not reliable across languages. |
NeuroSym-Cal: Bridging the Reasoning-Execution Gap in Code Generation via Hierarchical Calibration (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing calibration methods rely on the assumption that consensus implies correctness . Existing methods fail under systematic errors, leading to miscalibrated high-confidence predictions. |
| Approach: | They propose a hierarchical calibration framework that measures confidence at two levels . they propose sensitivity analysis to measure local curvature of deductive process . |
| Outcome: | The proposed framework de-saturates overconfident errors and improves selective generation performance on OOD benchmarks. |
Self-Consistency Boosts Calibration for Math Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing solutions for math reasoning tasks use semantic parsing or AST decoding, but performance can degrade dramatically even with slight changes to the questions. |
| Approach: | They propose three calibration methods based on self-consistency for math reasoning tasks. |
| Outcome: | The proposed methods bridge model confidence and accuracy better than existing methods based on p(True) or logit. |
Task Calibration: Calibrating Large Language Models on Inference Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive zero-shot performance on inference tasks, however, they may suffer from spurious correlations between input texts and output labels, which limits their ability to reason based purely on general language understanding. |
| Approach: | They propose a zero-shot and inference-only calibration method inspired by mutual information which recovers LLM performance through task reformulation. |
| Outcome: | The proposed calibration method improves on 13 benchmarks and prompt templates and can be integrated with other calibration methods. |
Relevance Scores Calibration for Ranked List Truncation via TMP Adapter (2025.findings-acl)
Copied to clipboard
| Challenge: | ranked list truncation methods struggle with limited capacity, unstable training and inconsistency of selected thresholds. |
| Approach: | They propose a new approach that incorporates the Threshold Margin Penalty as an additive loss function to calibrate ranking model relevance scores for ranked list truncation. |
| Outcome: | The proposed method improves on retrieval datasets and offers theoretical and practical benefits. |
DisCal: Distribution-Aware Calibration for Mathematical Reasoning Under Character-Level Noisy Inputs (2026.acl-long)
Copied to clipboard
Bo Zhang, Jiawei Zhang, Cong Gao, Bingxu Han, Minghao Hu, Jun Zhang, Yunbo Cao, Zhunchen Luo, Wen Yao, Guotong Geng, Zhong Wang
| Challenge: | Existing methods for calibration of large reasoning models (LRMs) focus on clean inputs, leaving noise unexplored. |
| Approach: | They propose a confidence calibration framework for character-level noisy inputs that extracts uncertainty signals from both the empirical answer distribution and the model’s predictive distribution and integrates them via a learned calibrator. |
| Outcome: | Experiments on multiple mathematical reasoning benchmarks show that DisCal outperforms existing calibration methods under noisy inputs, reducing expected calibration error (ECE) by up to 39.21% and improving Area Under the Receiver Operating Characteristic Curve (AUROC) by 31.44%. |
Calibrating Long-form Generations From Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Conventional calibration methods treat answer correctness as binary and do not work for long-form generation where an answer can be partially correct. |
| Approach: | They propose a framework where correctness of LLMs' responses and associated confidence levels are treated as distributions across a range of scores. |
| Outcome: | The proposed framework treats the correctness of the LLMs’ responses and their associated confidence levels as distributions across a range of scores. |
Fill In The Gaps: Model Calibration and Generalization with Synthetic Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing calibration methods negatively impact model accuracy due to the lack of diversity of validation data. |
| Approach: | They propose a calibration method that incorporates synthetic data without compromising accuracy. |
| Outcome: | The proposed method improves model accuracy on real data and reduces calibration error by 34% on four different tasks. |
Similarity-Distance-Magnitude Activations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for interpreting neural network-based language models (LMs) are limited to approximately conditional quantities. |
| Approach: | They introduce a similarity-distance-magnitude activation function and an SDM estimator to control class- and prediction-conditional accuracy among selective classifications. |
| Outcome: | The proposed estimator is more robust to covariate shifts and out-of-distribution inputs while remaining informative over in-difference data. |
Beyond Generation: Leveraging LLM Creativity to Overcome Label Bias in Classification (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to mitigate label bias by leveraging in-domain data are often unavailable in real-world scenarios. |
| Approach: | They propose a calibration method that generates synthetic in-domain data from a few in-context demonstrations and utilizes it for calibration. |
| Outcome: | The proposed method reduces label bias by leveraging in-domain data from demonstrations. |
KGE Calibrator: An Efficient Probability Calibration Method of Knowledge Graph Embedding Models for Trustworthy Link Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for probability calibration of knowledge graph embedding models are ill-suited for KGEs. |
| Approach: | They propose a method to calibrate knowledge graph embedding models for ranking-based link prediction using a Jump Selection Strategy and Multi-Binning Scaling to enhance reliability. |
| Outcome: | Experiments show that the KGEC outperforms existing calibration methods in terms of effectiveness and efficiency. |